Audio-visual approaches involving visual inputs have laid the foundation for recent progress in speech separation. However, the optimization of the concurrent usage of auditory and visual inputs is still an active research area. Inspired by the cortico-thalamo-cortical circuit, in which the sensory processing mechanisms of different modalities modulate one another via the non-lemniscal sensory thalamus, we propose a novel cortico-thalamo-cortical neural network (CTCNet) for audio-visual speech separation (AVSS). First, the CTCNet learns hierarchical auditory and visual representations in a bottom-up manner in separate auditory and visual subnetworks, mimicking the functions of the auditory and visual cortical areas. Then, inspired by the large number of connections between cortical regions and the thalamus, the model fuses the auditory and visual information in a thalamic subnetwork through top-down connections. Finally, the model transmits this fused information back to the auditory and visual subnetworks, and the above process is repeated several times. The results of experiments on three speech separation benchmark datasets show that CTCNet remarkably outperforms existing AVSS methods with considerablely fewer parameters. These results suggest that mimicking the anatomical connectome of the mammalian brain has great potential for advancing the development of deep neural networks. Project repo is https://github.com/JusperLee/CTCNet.

This is a brief technical report of our proposed method for Multiple-Object Tracking (MOT) Challenge in Complex Environments. In this paper, we treat the MOT task as a two-stage task including human detection and trajectory matching. Specifically, we designed an improved human detector and associated most of detection to guarantee the integrity of the motion trajectory. We also propose a location-wise matching matrix to obtain more accurate trace matching. Without any model merging, our method achieves 66.672 HOTA and 93.971 MOTA on the DanceTrack challenge dataset.

Adversarial attacks can easily fool object recognition systems based on deep neural networks (DNNs). Although many defense methods have been proposed in recent years, most of them can still be adaptively evaded. One reason for the weak adversarial robustness may be that DNNs are only supervised by category labels and do not have part-based inductive bias like the recognition process of humans. Inspired by a well-known theory in cognitive psychology -- recognition-by-components, we propose a novel object recognition model ROCK (Recognizing Object by Components with human prior Knowledge). It first segments parts of objects from images, then scores part segmentation results with predefined human prior knowledge, and finally outputs prediction based on the scores. The first stage of ROCK corresponds to the process of decomposing objects into parts in human vision. The second stage corresponds to the decision process of the human brain. ROCK shows better robustness than classical recognition models across various attack settings. These results encourage researchers to rethink the rationality of currently widely-used DNN-based object recognition models and explore the potential of part-based models, once important but recently ignored, for improving robustness.

It is well believed that the higher uncertainty in a word of the caption, the more inter-correlated context information is required to determine it. However, current image captioning methods usually consider the generation of all words in a sentence sequentially and equally. In this paper, we propose an uncertainty-aware image captioning framework, which parallelly and iteratively operates insertion of discontinuous candidate words between existing words from easy to difficult until converged. We hypothesize that high-uncertainty words in a sentence need more prior information to make a correct decision and should be produced at a later stage. The resulting non-autoregressive hierarchy makes the caption generation explainable and intuitive. Specifically, we utilize an image-conditioned bag-of-word model to measure the word uncertainty and apply a dynamic programming algorithm to construct the training pairs. During inference, we devise an uncertainty-adaptive parallel beam search technique that yields an empirically logarithmic time complexity. Extensive experiments on the MS COCO benchmark reveal that our approach outperforms the strong baseline and related methods on both captioning quality as well as decoding speed.

For saving cost, many deep neural networks (DNNs) are trained on third-party datasets downloaded from internet, which enables attacker to implant backdoor into DNNs. In 2D domain, inherent structures of different image formats are similar. Hence, backdoor attack designed for one image format will suite for others. However, when it comes to 3D world, there is a huge disparity among different 3D data structures. As a result, backdoor pattern designed for one certain 3D data structure will be disable for other data structures of the same 3D scene. Therefore, this paper designs a uniform backdoor pattern: NRBdoor (Noisy Rotation Backdoor) which is able to adapt for heterogeneous 3D data structures. Specifically, we start from the unit rotation and then search for the optimal pattern by noise generation and selection process. The proposed NRBdoor is natural and imperceptible, since rotation is a common operation which usually contains noise due to both the miss match between a pair of points and the sensor calibration error for real-world 3D scene. Extensive experiments on 3D mesh and point cloud show that the proposed NRBdoor achieves state-of-the-art performance, with negligible shape variation.

Although substantial efforts have been made using graph neural networks (GNNs) for AI-driven drug discovery (AIDD), effective molecular representation learning remains an open challenge, especially in the case of insufficient labeled molecules. Recent studies suggest that big GNN models pre-trained by self-supervised learning on unlabeled datasets enable better transfer performance in downstream molecular property prediction tasks. However, they often require large-scale datasets and considerable computational resources, which is time-consuming, computationally expensive, and environmentally unfriendly. To alleviate these limitations, we propose a novel pre-training model for molecular representation learning, Bi-branch Masked Graph Transformer Autoencoder (BatmanNet). BatmanNet features two tailored and complementary graph autoencoders to reconstruct the missing nodes and edges from a masked molecular graph. To our surprise, BatmanNet discovered that the highly masked proportion (60%) of the atoms and bonds achieved the best performance. We further propose an asymmetric graph-based encoder-decoder architecture for either nodes and edges, where a transformer-based encoder only takes the visible subset of nodes or edges, and a lightweight decoder reconstructs the original molecule from the latent representation and mask tokens. With this simple yet effective asymmetrical design, our BatmanNet can learn efficiently even from a much smaller-scale unlabeled molecular dataset to capture the underlying structural and semantic information, overcoming a major limitation of current deep neural networks for molecular representation learning. For instance, using only 250K unlabelled molecules as pre-training data, our BatmanNet with 2.575M parameters achieves a 0.5% improvement on the average AUC compared with the current state-of-the-art method with 100M parameters pre-trained on 11M molecules.

深度学习（DL）的快速增长和部署目睹了新兴的隐私和安全问题。为了减轻这些问题，已经讨论了安全的多方计算（MPC），以实现隐私保护DL计算。在实践中，它们通常是在很高的计算和沟通开销中，并有可能禁止其在大规模系统中的受欢迎程度。两种正交研究趋势吸引了人们对安全深度学习的能源效率的巨大兴趣，即MPC比较方案的高架降低和硬件加速度。但是，他们要么达到较低的减少比率，因此由于计算和通信节省有限而遭受了高潜伏期，或者是渴望的，因为现有的作品主要集中在CPU和GPU等一般计算平台上。在这项工作中，作为第一次尝试，我们通过将加密构件构建块的硬件延迟整合到DNN损耗功能中，以实现高能量效率，开发了一个系统的polympcnet，以减少MPC比较协议和硬件加速的联合额外降低的系统框架Polympcnet。和安全保证。我们的关键设计原理不是在DNN进行良好训练之后（通过删除或删除某些非物质操作员）训练（通过删除或删除某些非物质操作员）之后检查模型敏感性，而是要准确地执行DNN设计中的假设 - 培训DNN既是DNN都硬件有效且安全，同时逃脱了当地的最小值和鞍点并保持高精度。更具体地说，我们提出了通过多项式激活初始化方法直接提出的加密硬件友好的可训练多项式激活功能，以替代昂贵的2P-RELU操作员。我们开发了一个密码硬件调度程序和现场可编程门阵列（FPGA）平台的相应性能模型。

在本文中，我们提出了一个称为SDFE-LV的大规模，多源和不受约束的数据库，用于发现长视频中完整动态面部表达的发作和偏移帧，这被称为动态面部表情斑点的主题（DFE）和许多面部表达分析任务的重要步骤。具体而言，SDFE-LV由1,191个长视频组成，每个视频包含一个或多个完整的动态面部表情。此外，在相应的长视频中，每个完整的动态面部表达都被10次训练有素的注释者独立标记了五次。据我们所知，SDFE-LV是DFES任务的第一个无限制的大规模数据库，其长期视频是从多个现实世界/密切现实世界中的媒体来源收集的，例如电视采访，纪录片，电影和电影，以及我们媒体短视频。因此，在实践中，SDFE-LV数据库上的DFE任务将遇到许多困难，例如头部姿势变化，遮挡和照明。我们还通过使用许多最新的深度发现方法，从不同角度提供了全面的基准评估，因此对DFE感兴趣的研究人员可以快速而轻松地开始。最后，通过有关实验评估结果的深入讨论，我们试图指出几个有意义的方向来处理DFES任务，并希望将来DFE可以更好地进步。此外，SDFE-LV将仅尽快自由发布供学术使用。

大多数现有的语义分割方法都以图像级类标签作为监督，高度依赖于从标准分类网络生成的初始类激活图（CAM）。在本文中，提出了一种新颖的“渐进贴片学习”方法，以改善分类的局部细节提取，从而更好地覆盖整个对象的凸轮，而不仅仅是在常规分类模型中获得的CAM中的最歧视区域。 “补丁学习”将特征映射破坏成贴片，并在最终聚合之前并行独立处理每个本地贴片。这样的机制强迫网络从分散的歧视性本地部分中找到弱信息，从而提高了本地细节的敏感性。 “渐进的补丁学习”进一步将特征破坏和补丁学习扩展到多层粒度。与多阶段优化策略合作，这种“渐进的补丁学习”机制隐式地为模型提供了跨不同位置粒状性的特征提取能力。作为隐式多粒性渐进式融合方法的替代方案，我们还提出了一种明确的方法，以同时将单个模型中不同粒度的特征融合，从而进一步增强了完整对象覆盖的凸轮质量。我们提出的方法在Pascal VOC 2012数据集上取得了出色的性能，例如，测试集中有69.6 $％miou），它超过了大多数现有的弱监督语义细分方法。代码将在此处公开提供，https://github.com/tyroneli/ppl_wsss。

生成精确的类感知的伪基真实，也就是类激活图（CAM），对于弱监督的语义分割至关重要。原始CAM方法通常会产生不完整和不准确的定位图。为了解决这个问题，本文提出了基于可变形卷积中的偏移学习的扩展和收缩方案，以依次改善两个各个阶段中定位对象的回忆和精度。在扩展阶段，在可变形卷积层中的偏移学习分支，称为“扩展采样器”，寻求采样越来越小的判别对象区域，这是由逆监督信号驱动的，从而最大程度地提高了图像级分类损失。然后在收缩阶段逐渐将位置更完整的物体逐渐缩小到最终对象区域。在收缩阶段，引入了另一个可变形卷积层的偏移学习分支，称为“收缩采样器”，以排除在扩展阶段参加的假积极背景区域，以提高定位图的精度。我们在Pascal VOC 2012和MS Coco 2014上进行了各种实验，以很好地证明了我们方法比其他最先进的方法对弱监督语义分割的优越性。代码将在此处公开提供，https://github.com/tyroneli/esol_wsss。